feat: add tabular + columnar side indexes - #58

Storage-flavor boundary

WorkTable now has a clear three-flavor model:

1. Tabular — the existing authoritative row engine.
2. Tabular + columnar side indexes — this PR. Selected values and clustered-key metadata are duplicated into derived side structures for cheaper columnar-flavored access.
3. Columnar — an authoritative vector layout, vector execution, encoded/sealed segments, and native columnar persistence. Not covered by this PR.

This PR deliberately implements the low-cost middle flavor. It does not claim to be a full column store.

Summary

• add bare or configured per-field columnar syntax without introducing a table layout switch
• add columnar_indexes with one meaningful key declaration: cluster_by
• move table-wide settings to config.columnar_slot_id and config.columnar_chunk_rows
• support ColumnSlotId8|16|32|64, defaulting to 32 bits, without replacing the primary key
• duplicate opted-in values into mutable chunked side vectors and maintain ordered BTreeMap side-index metadata
• generate exact lookup, clustered-order scan, direct field scan, projection, dirty-state/rebuild, and capacity-observability methods
• maintain or invalidate the derived side state across insert, update, in-place update, delete, reinsert, and vacuum paths
• keep the current persistent format unchanged by rebuilding derived side state from authoritative rows after load

Identity and correctness

ColumnSlotId uses the full configured width only as a compact side-vector position. It is neither sort rank nor durable identity. ColumnarRowRef is opaque and validates:

authoritative primary key + slot + separate u64 generation + table incarnation

This closes the same-primary-key delete/reinsert ABA hole without reducing the configured live-slot capacity. Generation never wraps; an exhausted generation permanently retires that slot. The table incarnation rejects references from a different or reopened instance. Only primary_key() is public, and the reference is not serializable.

Choosing a slot width large enough for maximum simultaneously live columnar rows is the schema author's responsibility. Exhaustion returns WorkTableError::ColumnSlotIdExhausted(bits), rolls back the authoritative mutation, and never widens, truncates, wraps, evicts, or panics.

DSL corrections from expert review

• bare columnar uses the table chunk default
• columns: [...] inside a side index is rejected as semantically redundant
• include is reserved and rejected until covering projection exists
• only compression(none) is accepted; inert auto/delta/rle/dictionary declarations are errors
• per-field chunk overrides must be power-of-two multiples/divisors of the table default
• primary-key columnar, unknown/non-columnar cluster keys, duplicate config, method-name collisions, unknown postfix attributes, and unsupported worktable_version! syntax are rejected

Current implementation boundary

• the row engine and primary key remain authoritative
• side fields currently use mutable Vec<Vec<Option<T>>>; no SIMD/vector kernel, validity bitmap, string arena, codec, or native side-index disk format is claimed
• generated scans/projections return owned Vec values and materialize the result
• exact clustered lookup currently requires the complete composite key; prefix/range builders and scan_batches are follow-ups
• one table-local columnar RwLock gives each operation a coherent side-index snapshot but serializes concurrent side-index writers
• fallback in-place paths mark the replica dirty; columnar_is_dirty() and rebuild_columnar() let applications schedule the current whole-table rebuild cost

SAP HANA comparison

The v3 guide compares this work with Sikka et al., Efficient Transaction Processing in SAP HANA Database: The End of a Column Store Myth (SIGMOD 2012). HANA's L1 row delta → L2 column delta → compressed main lifecycle is precedent for a future full-columnar flavor. The honest analogy here is narrower: WorkTable retains its tabular engine and adds optional uncompressed side indexes; it does not implement HANA's L2/main lifecycle or vector execution.

Validation

• cargo test --all-features: 171 library tests passed; 524 integration tests passed, 5 intentionally ignored; 2 doctests passed
• focused side-index tests cover mutation, exact/ordered access, projection, dirty rebuild, slot exhaustion rollback, different-primary-key and same-primary-key slot reuse, table-incarnation isolation, and concurrent reinsert/refresh
• direct Using coverage passes with WorkTablesIndex, congee, and arctic primary backends
• cargo clippy --all-targets --all-features -- -D warnings
• cargo check --no-default-features
• cargo fmt --all -- --check

Documents

• docs/columnar-index-plan.md — implementation plan and scope boundary
• docs/columnar-fields-and-indexes-guide-v3.md — complete technical/reviewer guide, including the HANA comparison
• output/pdf/worktable-columnar-side-indexes-guide-v3.pdf — visually reviewed 9-page PDF

Performance claims remain gated on the wt-benchmarks matrix: tabular baseline vs fields-only vs fields-plus-clustered side indexes, row-path overhead, p50/p95/p99, concurrency, allocations, memory, code size, dirty rebuild, persisted reload, and all supported Using backends.